Repository navigation
ci(agent-e2e): run on gpt-6-luna; skip live dall-e-3/tts-1 tests - #191
Merged
Merged
Conversation
The OpenAI key is moving to the Orkes-Developers project, which only allows the gpt-5.6 / gpt-6 families. gpt-4o-mini also shuts down on 2027-01-14. - Set CONDUCTOR_AGENT_LLM_MODEL=openai/gpt-6-luna for every suite. - Suite 11 (LangGraph): build ChatOpenAI from that model, without temperature. Reasoning models reject temperature unless reasoning effort is "none". - Suite 7: skip the live DALL-E 3 and TTS-1 tests. dall-e-3 no longer exists at OpenAI, gpt-image-* is not enabled, and the project gets a 403 for tts-1. - Add an "LLM provider check" first step that calls the model directly, so a dead key or quota fails in seconds, not at the 45-minute timeout. - Add a "Show server errors" step that prints server.log errors to the job log on failure or cancel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert before merge. A server bisect: on 3.34.0-rc2, test_hitl_approve_path stalled twice in the same place, and guardrail matrix #9 timed out once. Run only those two tests, in a matrix against the org-var server version and 3.33.0, the last server the full JS suite passed on (2026-10-26). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert before merge. test_hitl_approve_path stays RUNNING after one approval on both 3.34.0-rc2 and 3.33.0 with gpt-6-luna, but passed on 3.33.0 with gpt-4o-mini. After the wait, print every task in its workflow, with full inputs and outputs for the LLM and HUMAN tasks, to see what the model does after the approved publish_article call. Run only that test, still on both servers. Guardrail matrix #9 passed alone on both and goes back to normal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On gpt-6-luna, test_hitl_approve_path stalled on every run (3.33.0 and 3.34.0-rc2). After the approved publish_article call, the server sends the instructions and the original user request again, after the tool result. Read literally, that is a new request: luna wrote and published a second article, which waits for a second approval the test never gives. gpt-4o-mini treated the request as already done. Keep the task in the user message only. Make the instructions say how to do the work and that one successful publish_article ends the request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert before merge. Make the test's system prompt, tool description
and user message overridable (HITL_SYSTEM, HITL_TOOL_DESC, HITL_USER).
Run one matrix job per variant on the org-var server (3.34.0-rc2), in
place of the 3.33.0 vs rc2 legs. Both servers behaved the same.
- control: current text
- user-stop: stop rule in the user message, which the server
re-sends after the tool results each turn
- tool-once: tool description says it can be called at most once
per chat session
- system-count: system prompt says to count publish_article calls and
end the session when the count is above zero
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert before merge. The control prompt failed in 4 of 4 runs. Keep the three passing variants (user-stop, tool-once, system-count) for repeat runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the temporary bisect, diagnostic and prompt-variant changes: the workflow and suites go back to their state at 3379c12, and the full suite runs again. On gpt-6-luna, test_hitl_approve_path stalled on both 3.33.0 and 3.34.0-rc2. After the approved publish_article call, the server sends the original user request again, after the tool results. Luna often reads that as a new request and publishes a second article, which waits for a second approval the test never gives. Prompt variants tested in CI (passed / runs): - "publish once; when it returns 'published' you are done": 0/4 - tool description "at most once per chat session": 2/4 - user message "if already published, do not publish again": 3/4 - system prompt: count earlier calls, stop if above zero: 4/4 Use the last one as the agent's instructions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chrishagglund-ship-it
requested review from
NicholasDCole,
kowser-orkes and
ling-senpeng13
November 13, 2026 16:27
The live image and TTS tests now run by default and are skipped only when SKIP_IMAGE_GEN / SKIP_TTS is "true". agent-e2e.yml sets them, because the Orkes-Developers OpenAI project has no image or TTS models. Also keep "Show server errors" green when grep finds no match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ling-senpeng13
approved these changes
Nov 16, 2026
ling-senpeng13
left a comment
Contributor
There was a problem hiding this comment.
Looks good — approving.
- Suite 16 HITL test (optional, but I'd recommend it): I think the real problem is the test itself, not the model. It approves only once, so it only passes if the model calls the tool exactly once. Even the same model won't always do that, since its output varies from run to run, so this test can turn flaky again later. That's also why the prompt had to get so specific, and I'd rather not rely on a prompt like that: it will likely break again with the next model. If the test approved every time it was asked, it wouldn't depend on how the model behaves, and the prompt could go back to normal.
- Nit:
e2e/helpers.tsstill defaultsMODELtogpt-4o-mini, which shuts down in January.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The OpenAI key is moving to the Orkes-Developers project, which only allows the gpt-5.6 / gpt-6 families. gpt-4o-mini also shuts down on 2027-01-14.